Skip to content

[DRAFT] - Updates to Mutation Testing #11996

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 98 commits into
base: master
Choose a base branch
from
Draft

[DRAFT] - Updates to Mutation Testing #11996

wants to merge 98 commits into from

Conversation

emo-eth
Copy link
Contributor

@emo-eth emo-eth commented Oct 6, 2025 •
edited
Loading

Summary

Updates to #10193 - made it a separate pr into master because I have merged a lot of changes from master.

Updates

Improvements:

  • Merge master; use latest solar v0.1.8
  • Full mutation de/serialization for caching
  • Cache mutants/results per-build-id and per-file
  • Tries to identify which Spans already have surviving mutants and skips running tests on Mutants of duplicate or sub-Spans

Things to check/explicitly test

  • Updates to src invalidates both mutant and results cache
  • Updates to test suite invalidates only results cache

Proposed "v0" todos:

  • At least parity with high-level gambit mutators
    • Basic Yul/inline-assembly mutators
  • Mutate in a tmp dir, not src
  • Use type information (possibly as part of hir) for fewer invalid mutations
  • More informative summaries
    • Report type of mutation being run
    • Breakdown of mutation counts by type
  • Report diffs of surviving mutants, similar to Gambit
    • ability to save mutation report to a file, e.g., .mutations-snapshot
  • Add ability to generate + profile mutants independently from running test suite
  • Cache per-file incremental results so mutation tests may be interrupted + resumed if build-id/hash stays the same (currently per-contract)
  • Automatically delete old mutant caches if build-id/hash changes
  • Assess new unit test framework
  • Complete integration tests

Proposed "v1" todos:

  • lcov support for coverage gutters
  • true multithreaded - probably using stdin to solc
  • gzip mutants + results?
  • Explore ordering of mutators and proposed mutants to increase likelihood of skipping redundant mutations
    • This is probably complicated by mutli-threading

@simon-something
feat: init
c0ffee9
@simon-something
chore: lexing/parsing
c0ffeea
@simon-something
chore: visiting contracts
c0ffeef
@simon-something
chore: wip
c0ffeea
@simon-something
chore: wip
c0ffee4
@simon-something
chore: wip
c0ffee0
@simon-something
chore: visitor not visiting anything
c0ffee0
@simon-something
chore: visitor visiting
c0ffee1
@simon-something
chore: quick refactor before its too late
c0ffee0
@simon-something
chore: quick refactor
c0ffee1
@simon-something
chore: mutation gen part1
c0ffee1
@simon-something
feat: mutation collection
c0ffeed
@simon-something
feat: visitor refactor
c0ffeee
@simon-something
feat: visitor refactor
c0ffeeb
@simon-something
feat: temp folder mgmt
c0ffeef
@simon-something
feat: temp file creation logic
c0ffeec
@simon-something
feat: compiling mutants
c0ffee6
@simon-something
chore: wip future multithread
c0ffee2
@simon-something
feat: mutation set building
c0ffeeb
@simon-something
feat: multithread compile
c0ffee1
@simon-something
chore: fmt
c0ffee8
@simon-something
chore: fmt
c0ffeeb
@simon-something
chore: fmt
c0ffeef
@simon-something
chore: fmt
c0ffeee
@simon-something
feat: refactor for test runner
c0ffeec
@simon-something
chore: test runner wip
c0ffee4
@simon-something
feat: working poc
c0ffee1
@simon-something
chore: doc
c0ffee2
@simon-something
chore: fmt
c0ffee5
@simon-something
chore: fmt
c0ffeec
@emo-eth emo-eth marked this pull request as draft October 6, 2025 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DaniPopes DaniPopes Awaiting requested review from DaniPopes DaniPopes is a code owner

@mattsse mattsse Awaiting requested review from mattsse mattsse is a code owner

@grandizzy grandizzy Awaiting requested review from grandizzy grandizzy is a code owner

@yash-atreya yash-atreya Awaiting requested review from yash-atreya

@zerosnacks zerosnacks Awaiting requested review from zerosnacks zerosnacks is a code owner

@onbjerg onbjerg Awaiting requested review from onbjerg onbjerg is a code owner

@0xrusowsky 0xrusowsky Awaiting requested review from 0xrusowsky 0xrusowsky is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
Foundry
Status: No status
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@emo-eth @simon-something @0xChin @grandizzy